Summary

Implements #75810

This changes _install_package() to return early when it detects that the package might be in the process of being installed by another process.

It does so by checking the install_status and install_started_at fields on the epm-packages saved object. If the previous installation was started longer than MAX_TIME_COMPLETE_INSTALL (currently, 60 seconds) ago, the installation will be attempted again.

MAX_TIME_COMPLETE_INSTALL is also used by ensurePackagesCompletedInstall() to decide whether to reattempt package installations. This should not break, or be broken by, the change made in this PR.

How to test this

I've created https://gist.github.com/skh/cc695952031c9e349874b898c7066e42 to test this locally. On my system, with a local ES instance

• when the two installation attempts are about 2-5 seconds apart, the behavior is as expected: the second installation attempt returns early and lists the assets that have been installed by the first attempt up until then. That means the second attempt lists less assets in its response than the first.
• when the two installation attempts are more than 10 seconds apart, the first one completes before the second is started, so two complete installations are performed. Both attempts list the same number of assets in their responses.
• when the two installation attempts are really close together, we still run into a race condition with the saved object itself. I can trigger this by sending the two install attempts within about 500 milliseconds of each other:

Response for second install attempt: {"statusCode":404,"error":"Not Found","message":"Saved object [epm-packages/apache] not found"}
Response for first install attempt: {"statusCode":409,"error":"Conflict","message":"[epm-packages:apache]: version conflict, required seqNo [1879], primary term [1]. current document has seqNo [1880] and primary term [1]: version_conflict_engine_exception"}
Starting package delete
Response for delete attempt: {"statusCode":400,"error":"Bad Request","message":"apache is not installed"}

Fixing this is beyond the scope of this PR.

UPDATE: after discussing with the Kibana Core team, the 409 error above happens when the saved object version has changed between a read and a subsequent update operation. We can use SavedObjectsClient.errors.isConflictError() to catch this specific case, and repeat the check for installed_status and install_started_at.

Follow-up issues: #84651 and #84656